Skip to content

chore: fix prompt-audit findings in knowledge through pixel-art - #16334

Merged
kyle-sexton merged 3 commits into
mainfrom
chore/prompt-audit-knowledge-to-pixel-art
Nov 15, 2026
Merged

kyle-sexton merged 3 commits into
mainfrom
chore/prompt-audit-knowledge-to-pixel-art

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

No related issue: prompt-audit findings from an owner-requested full-read audit

Summary

Group 6 of 10 from a full-read prompt audit, covering 11 plugins: knowledge, machine-health, markdown-format, mcp-tools, miro, multi-agent, mutation-testing, naming, overengineering, performance and pixel-art. The audit found instruction text written for older models, stale against the repository, or contradicted by another file. A second agent verified every edit before it reached this branch.

The change is instruction text except in six code files, where only a string changes:

  • plugins/knowledge/workflows/docpage-digest-batch.js and plugins/multi-agent/workflows/drift-audit.js: one prompt string each, capital-letter emphasis lower-cased.
  • plugins/miro/server/src/tools/bulk.ts and tags.ts: MCP tool descriptions now state the failure mode and return shape the handlers already have.
  • plugins/performance/scripts/discriminate.py and spawn-census.sh: refusal messages drop the story of an earlier run and keep the rule, its reason and the fix.

Four eval files change: three mcp-tools grader criteria drop a trailing parenthetical that repeated the criterion, and the multi-agent su-apply-guard-off-confirms expected outcome drops a local-layer offer that its own description says cannot arise at a home root (no grader checks it).

Fix

  • Capital-letter emphasis lower-cased per ADR 0098 (146 of the 281 hunks change case only; heading anchors are unchanged).
  • Facts corrected against the code: machine-health's severity rubric now matches Test-Defender.ps1 (tamper protection off is WARN; a detection is CRIT only without a recorded remediation), the remediation policy names the disk-space check and the first-run dry run, and the audit skill says the orchestrator writes the report and state files. course-digest's screenshot paths and referer header, and video-digest's key-frame folder and phase number, match the extractors.
  • Restated upstream facts replaced by pointers (Chrome remote debugging, ffmpeg, the plugin variables, subagent model values, the effort cap, plugin auto-update, gh and benchmark tools).
  • History and provenance notes removed. The miro setup boundary now names its real gate (verify-api, read-only call only) instead of an undefined "explicit confirmation", which narrows the rule.

No hunk was dropped or reverted: the patch applied cleanly on main, and no edit loosens a prohibition or safety rule. The pr-refine/fix-docs lane added b52d695, replacing three figures of speech in overengineering and performance docs with plain wording.

Verification

  • git apply --3way of the group patch: all 86 files applied cleanly on 99c790c93c.
  • plugins/performance/scripts/discriminate.test.sh (38 pass) and spawn-census.test.sh (30 pass), which assert on the edited messages.
  • scripts/sync-shared-copies.sh --check: all 904 copies match. scripts/check-changelog-fragments.sh --check and --check-required origin/main: 11 valid fragments, one per touched plugin.
  • markdownlint-cli2 over the 11 plugins and the fragments (374 files, 0 issues); biome format on the two miro files (fixed one quote style); pinned ruff on discriminate.py.
  • Left out: the miro vitest suite (its tests assert no description text) and the eval suites (no paid run for wording edits).

Related

N/A

Follow-ups

  • Pre-existing (advisory): history notes the audit did not touch remain in plugins/performance/scripts/README.md L55 ("three of the five source-run failures") and plugins/performance/scripts/spawn-census.sh L223 ("source failures 3 and 4"), per the claude-review lane's observation on b52d695; they do not affect behavior.

🤖 Generated with Claude Code

kyle-sexton and others added 2 commits November 15, 2026 08:49
Apply the verified prompt-audit edits for the knowledge, machine-health,
markdown-format, mcp-tools, miro, multi-agent, mutation-testing, naming,
overengineering, performance and pixel-art plugins: capital-letter
emphasis lower-cased (ADR 0098), stale paths and facts corrected against
the code, restated upstream facts replaced by pointers, and history
notes removed. Code files change only in prompt, message or tool
description strings.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The typos check reads "PNGs" as a misspelling; say "images" instead.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 15, 2026 13:58
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-15T14:00:13.583201Z 8ea8740 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-refine/fix-docs
Model rung 0
Leg handoff
Run job success
Activity step success
Commits pushed none (read lane)
Run 43721857422

Failed GitHub API calls

Method Path Status
GET /repos/{repo}/git/ref/tags/* 404

@github-actions

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-refine/apply-patch
Model rung 0
Leg base
Run job success
Activity step success
Commits pushed b52d695
Run 43721857422

Failed GitHub API calls

Method Path Status
GET /repos/{repo}/git/ref/tags/* 404

@github-actions

Copy link
Copy Markdown
Contributor

claude-security-review reviewed b52d695cfbed59a5ef5cf2717a2d6d35e45b9c4d: read 97 of 97 changed file(s); 0 finding(s), 0 as inline comments.

Chunk 1 of 3 (40 files)

Security review: chunk 1 of 3 (40 files)

No security issues found in this chunk.

I read every hunk of all 40 files: the 11 .changes/ fragments, the plugins/knowledge instruction files, and plugins/knowledge/workflows/docpage-digest-batch.js. None of them is on the trusted-path list. The chunk changes no plugin manifest, workflow, action, token request or merge-hold file.

What I checked:

  • Untrusted-content fencing in docpage-digest-batch.js. Only the closing line of fence() changes, from "DATA" to "data". The escaping that stops a fenced block from being closed early (.replace(/</g, '\\u003c')) is unchanged, and so are the UNTRUSTED rule text and the <data> wrapper. The framing still works the same way.
  • Instruction-surface lens: no rule was removed or loosened.
    • distill/SKILL.md (untrusted EPUB handling): only the capital letters change, in ABORT, ANY and INSIDE. Every safety step is still there: refuse absolute or .. entry paths, delete symlinks, and require each full-path or href to resolve inside the cache directory.
    • docpage-digest/SKILL.md: the canonical-URL rules and the isPublicUrl requirement are unchanged.
    • video-digest/SKILL.md "Spoke paths": the guard still says to accept an inherited CLAUDE_PLUGIN_DATA only when it names this plugin. The edit only replaces the dated basis with a pointer.
    • anthropic-docs-profile.md: the rule that retagging runs only inside a graduation-time verification cycle now appears once, at the top, instead of three times. That rule is not a security control.
    • map-corpus and inventory-format: only history notes are removed. The gates still fail loudly on bad input.
    • Protected-class register: I didn't consult the register's list because no rule was removed or loosened, so there was nothing to check against it.
  • Credential and auth text. course-digest/SKILL.md drops only a "NOTE:" prefix. Credentials still come from environment variables and are not stored in userConfig. The referer change in screenshot-strategy.md is documentation only: it describes the platformConfig.referer value the extractor already sends.
  • New external pointers. The ffmpeg docs, the Chrome remote-debugging blog post and yt-dlp's twitter.py on GitHub are added as pages to read, not code to run. They add no new execution path.

Spend: none. No change to turn limits, models, schedules or runs.

Chunk 2 of 3 (40 files)

Security review: chunk 2 of 3 (40 files: machine-health through overengineering/findings-artifact.md)

No security issues found in this chunk.

I read every hunk of all 40 files. Almost all of the change is instruction text: emphasis lower-cased, upstream facts swapped for pointers, and doc facts aligned with the code. No plugin.json, marketplace.json, workflow, trust-control or trusted-path file is in this chunk, so the manifest and trust-control checks do not apply.

Instruction-surface deletions and changes I checked

The protected-class register was not read; each removed or changed rule was judged by its consequence.

  • plugins/miro/skills/setup/SKILL.md:151
    • Before: "Do not make a Miro API request without explicit confirmation."
    • After: a request is allowed only when the invocation passed verify-api, and only the read-only call in Task step 5.
    • This narrows the rule. Lines 2, 20 and 56 of the same file make verify-api the explicit opt-in that allows one read-only call. The next line still bans mutating Miro tools during setup.
  • plugins/mutation-testing/skills/audit/context/persist-findings.md
    • Removed: the paragraph saying the guard's .gitignore is the only write outside the findings file, and that step 3 proves it.
    • The constraint still stands in the same file: line 48 ("at most two writes"), lines 51–52, step 3 at line 111 and line 355. This is a compression, not a loosening.
  • plugins/machine-health/skills/audit/scripts/linux/not-implemented.md: the ban on sudo -n and on interactive prompts is kept. Only the stated reason changed.
  • Prohibitions in machine-health's Linux and macOS not-implemented.md, mcp-tools/audit-posture's config-read rule, and the setup explicit-decision rules: only the case changed. What they forbid is the same.
  • machine-health remediation-policy.md
    • The Clear-TempFiles trigger and abort condition now name the disk-space check instead of "storage on C:".
    • The remediation still defaults to DISABLED until the owner approves it, and the first run is still a dry run.
    • This is a doc alignment with no new unapproved action.
  • machine-health severity-rubric.md: tamper protection off moves from CRIT to WARN. The PR says this matches Test-Defender.ps1. It is report wording on the user's own workstation and has no trust-boundary impact.
  • machine-health audit/SKILL.md: the report and state files are now written by the orchestrator only. This narrows what the session writes.

Code files in this chunk

  • plugins/miro/server/src/tools/bulk.ts:29 and tags.ts:45: only the MCP tool-description strings change. No handler logic changes.
  • plugins/multi-agent/workflows/drift-audit.js:687: one prompt word lower-cased ("REFUTE" → "refute").

Evals

  • Three mcp-tools grader criteria lose a trailing "legacy expectation" parenthetical. The pass/fail conditions are unchanged.
  • The multi-agent su-apply-guard-off-confirms expected outcome drops a local-layer offer. The frontier-guard confirmation it checks is still required.

PR context

Nothing in the PR description or the prior-findings file tells the lane what to do. Prior findings: no earlier heads were reviewed, so this was not a blocking-only review.

Chunk 3 of 3 (17 files)

Security review, chunk 3 of 3 (17 files)

No security issues found in this chunk.

I read every hunk of all 17 files: the overengineering, performance and pixel-art plugin docs and skill bodies, plus plugins/performance/scripts/discriminate.py and spawn-census.sh.

  • Code files. discriminate.py and spawn-census.sh change only the text of a refusal message. The exit status (2 / harness_die), the control flow, the inputs and the interpolation are unchanged. The edited text in spawn-census.sh still uses only $tool and ${real:-<nothing>} inside the same double-quoted argument, so it adds no new expansion or injection surface.
  • Instruction-surface lens. The edits lower-case emphasis words (e.g. STOP → Stop, AND → and, IF/THEN → If/then) and remove notes about earlier runs. None of them drops what a rule forbids. The /performance:goal stop rules, the pixel-art adapter selection rule, the /performance:verify mode-coverage rule and the overengineering rollback ladder all keep the same conditions. The /performance:verify verifier-count sentence changes from "one verifier is the floor" to "One verifier is required; dispatch a second when…", which still requires one verifier. No rule in a protected class is narrowed, so no exception-register match. I did not consult docs/contracts/instruction-exception-register/README.md (it was not read in this chunk); I judged the edits by what each rule prevents.
  • New links. The added URLs (code.claude.com, cli.github.com, docs.github.com, pyperf, criterion, pkg.go.dev, the openjdk/jmh GitHub page) are reading pointers in markdown. Nothing fetches or runs them automatically.
  • Manifests and trusted paths. No plugin.json or marketplace.json change is in this chunk, and no file here matches a trusted-path glob.
  • PR context. The description contains no embedded instructions aimed at the review lane.

critical-findings: 0

@github-actions

Copy link
Copy Markdown
Contributor

claude-review reviewed b52d695cfbed59a5ef5cf2717a2d6d35e45b9c4d: read 97 of 97 changed file(s); 0 finding(s), 0 as inline comments.

Chunk 1 of 3 (40 files)

Code review: chunk 1 of 3 (40 files)

No findings. I read every hunk of the 40 chunk-1 files: the 11 changelog fragments, the knowledge plugin's instruction docs, and plugins/knowledge/workflows/docpage-digest-batch.js.

Most hunks only lower-case capital-letter emphasis or remove history notes. I checked that none of them changes what a rule requires: each prohibition and each must/never keeps its force, and no heading anchor changes. In anthropic-docs-profile.md, three copies of the rule "a retag of an already-verified slice runs inside a graduation-time verification cycle" become one statement at the top of the filter section. That statement covers every rule below it, so the rule now reaches the same cases or more.

Factual corrections checked against the code at head:

  • Screenshot names and folder (storage-schema.md): confirmed. extract-course-run.js:80 writes into screenshots/scene-01. vendor/video-digestion/frames/scene-detect.js:339 names scene frames scene_%04d.png, and lib/players/hotmart.js:173 names fallback frames interval_NNNN.png.
  • Referer header (screenshot-strategy.md): confirmed. extract-course.js:183 reads platformCfg.referer ?? "".
  • Phase number (watch-checklist.md 3.1): confirmed. watch-pipeline.md puts the claim-inventory format under ## Phase 3: claim inventory.
  • No script counts the reads (watch-pipeline.md): confirmed. watching/read-policy.js picks the resolution for each frame and keeps no count or threshold, so the old wording was the stale one.
  • No wait (youtube.md): confirmed. caption-wait.js:70-75 returns state: "none" for the cases described.
  • -ss placement (dometrain.md): confirmed. The verified command at line 119 does put -ss after -i.
  • Option A and Option B (multimodal-evaluation.md): both names point at sections that exist.
  • Key-frame folder (synthesis-contract.md): the new sentence matches the layout table's key-frames/frames/ row. No other video-digest doc still mentions a synthesis/ image folder.
  • Prompt string (docpage-digest-batch.js:555): only its case changes, and no test asserts on the old text.

Changelog fragments: all 11 declare bump: patch. That fits, since the changes are wording and doc corrections with no behavior change. No .claude-plugin/plugin.json or marketplace.json change is in this chunk, so no manifest lines apply.

PR context: the PR closes no issue, so there are no acceptance criteria to check. Its description contains no embedded instructions aimed at this review.

The verification the PR describes (test-suite passes, markdownlint, the sync and changelog checks) is the author's claim. I could not run any of it here.

Verifier dropped 0 of 0 candidates.

Chunk 2 of 3 (40 files)

Chunk 2 of 3: no findings. I read every hunk of all 40 files in this chunk: machine-health, markdown-format, mcp-tools, miro, multi-agent, mutation-testing, naming and overengineering/context/findings-artifact.md. This is the first review of this pull request, so it covers all findings, not only blocking ones.

Most hunks only lower-case emphasis or headings, and none of them changes a rule's meaning. Every heading whose text changed keeps its anchor; for example, #deliberately-not-type-review-findings is unchanged because anchors are lower-cased anyway.

I checked each change of fact against the code it describes:

  • Severity rubric. The defender rules now match Test-Defender.ps1:
    • tamper protection off is WARN (L123-130)
    • a detection is CRIT only with no recorded remediation (L57)
    • the window is 30 days ($recentDays = 30, L32)
  • Remediation policy (machine-health).
    • The check id disk-space exists (catalog/checks.jsonc).
    • "Any fixed volume under 15% free" matches the WARN/CRIT thresholds in Test-DiskHealth.ps1 L75.
    • first-run forces a dry run (Invoke-MachineHealthCheck.ps1 L147).
  • Report and state writes. The orchestrator writes the report, latest.json and history.jsonl (Invoke-MachineHealthCheck.ps1 L758-768), which matches the new audit steps 6 and 7.
  • succeeded field. The rename matches the remediation scripts and the schema in output-schema.md L166-172.
  • Shipped TODO.md. The new wording in approvals.md matches skills/audit/TODO.md, which only points to approvals.md and remediation-policy.md.
  • Miro tool descriptions.
    • bulk.ts: a failed createStickyNoteItem in the loop throws, so the earlier IDs are lost, as the description now says.
    • tags.ts: the stated return shape and the error for unsupported item types match the handler (L57-61).
  • Miro setup boundary. It now names verify-api and Task step 5, which exist in the skill.
  • Relative link. The new link in scripts/linux/not-implemented.md points to an existing file and heading (#a-note-on-sudo).
  • Removed passages (mutation-testing).
    • The paragraph cut from persist-findings.md is still covered by the step 3 and step 5 bullets above it.
    • The sentence cut from the top of restoration-regimes.md still appears at L15 of that file and in SKILL.md L218.
  • Eval change (multi-agent). The su-apply-guard-off-confirms expected outcome drops the local-layer offer. That eval's case.yaml has no grader that refers to it.

These checks come from reading the code. I did not run anything; the PR's own test results (discriminate.test.sh, spawn-census.test.sh, markdownlint) are the author's claims.

The chunk does not change any plugin manifest, and no .claude/rules/ file applies here. Spend: none.

Verifier dropped 0 of 0 candidates.

Chunk 3 of 3 (17 files)

Code review: chunk 3 of 3 (overengineering, performance, pixel-art)

I read all 17 files in this chunk, every hunk. I found no blocking or advisory findings.

What changed in this chunk

  • overengineering (4 files): two history sentences are cut to the rule they support, and capital-letter emphasis is lower-cased. Changing the heading PREVIOUS to previous leaves its anchor unchanged, because GitHub lower-cases anchors anyway, so the TOC link in baseline-model.md still resolves.
  • performance scripts: discriminate.py and spawn-census.sh change only the text of their refusal messages. Exit codes and control flow are the same. The tests check these messages only for substrings that the new text keeps:
    • discriminate.test.sh checks for HARNESS BROKEN and exited 127.
    • spawn-census.test.sh checks for undercounts.
    • The PR says both test suites pass. That is the author's claim; I did not run them.
  • performance docs/skills:
    • Upstream facts are replaced by links: the auto-update default, gh run view, the REST timing endpoint, and the Criterion, benchstat and JMH docs.
    • Emphasis is lower-cased.
    • verify/SKILL.md changes "one verifier is the floor, not the target" to "one required; a second when a missed defect would cost more than its run". That keeps the required minimum and makes the advice about a second verifier concrete. It loosens no requirement.
  • pixel-art:
    • Emphasis is lower-cased.
    • The Playwright aside is dropped.
    • animate and vfx still send motion review to preview.png, so the behaviour is unchanged.

Observation (not counted as a finding): some source-run history lines were left in place, and they sit next to edited lines. Examples are scripts/README.md#L55 ("three of the five source-run failures") and spawn-census.sh#L223 ("source failures 3 and 4"). These lines are unchanged and do not affect correctness. They only mean the provenance cleanup in these two files is incomplete.

Closing issue: none. The PR context says the work comes from an owner-requested audit, so there are no acceptance criteria to check against.

Spend: none. Nothing in this chunk touches billing, runners, or lane budgets.

Verifier dropped 0 of 0 candidates.

critical-findings: 0

@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 15, 2026
Merged via the queue into main with commit a4c9511 Nov 15, 2026
72 checks passed
@kyle-sexton
kyle-sexton deleted the chore/prompt-audit-knowledge-to-pixel-art branch November 15, 2026 14:44
@github-actions github-actions Bot added the merge-path: owner-agent Post-merge accounting record (README.md, Merge paths) label Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant